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Abstract. We address the important question of the extent to which 
random variables and vectors with truncated power tails retain the char- 
acteristic features of random variables and vectors with power tails. We 
define two truncation regimes, soft truncation regime and hard trun- 
cation regime, and show that, in the soft truncation regime, truncated 
power tails behave, in important respects, as if no truncation took place. 
On the other hand, in the hard truncation regime much of "heavy tailed- 
ness" is lost. We show how to estimate consistently the tail exponent 
when the tails are truncated, and suggest statistical tests to decide on 
whether the truncation is soft or hard. Finally, we apply our methods 
to two recent data sets arising from computer networks. 



1. Introduction 



Probability laws with power tails are ubiquitous in applications. A good 
fit between empirical distribution of various quantities of interest and distri- 
butions with power tails has been reported in such diverse areas as human 



Corral 


( 


2006)), ani- 


;uage (Serrano et al. 



(2009)). It is also true that m many situations there is a "physical" limit 



that prevents a quantity of interest from taking an arbitrarily large value. 
The File Allocation Table (FAT) used on most computer systems allows the 



largest file size to be 4GB (minus one byte) (Microsoft Knowledge Base Ar- 



ticle 154997 (2007)); the greatest loss an insurance company is exposed to by 



a single covered event is limited by its reinsurance contract (see e.g. Mikosch 
(2009;)). Even the number of the atoms in the universe is widely considered 



to be finite. It is common in practice to combine these two facts together 
and use a model that features power tails only in a truncated form; such 
models are often referred to as truncated Levy flights, see e.g. Scholtz and 
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Contreras (1998), Maruyama and Murakami (2003) or Zaninetti and Ferraro 



(2008). At the first glance this leads to a situation where the power tails, in 



a sense, completely disappear. The truncation may change dramatically the 
behavior of the cumulative sums of observations and it always changes dra- 
matically the behavior of the cumulative maxima of the observations. Yet 
it is precisely such patterns of behavior for which a model with power tails 
is chosen in the first place. This leads one to ask the natural question: to 
what extent, if any, do phenomena well described by models with 
truncated power tails retain the characteristic features of power 
tails? 

Answering this question is not straightforward. We start by pointing out 
that the level of truncation is linked to the amount of observations one has 
at hand. This can be thought of in different ways. First of all, finiteness 
of the sample is sometimes taken as the source of the truncation, see e.g. 



Burrooughs and Tebbens (2001) or Barthelemy et al. (2008). Secondly, both 



the physical nature of the truncation bound and the available data can be 
linked to a technological level. This is particularly transparent when one 
models a phenomenon related to computer or communications systems; see 
e.g. Jelenkovic (1999) or Gomez et al. (2000). We describe this situation as 



a sequence of models, each one with truncated power tails or, in other words, 
as a triangular array system, which we now proceed to define formally. 

Let F be a probability law on M. d , d > 1, with the following property. 
There exists a sequence (b n ) with b n f oo and a non-null Radon measure \i 
on? \ {0} with fi{W \ R d } = 0, such that 



(1.1) 



nF(b^- 



m(-) 



vaguely in M d \ {0}. Here M. d is the compactification of M d obtained by 
adding to the latter a ball of infinite radius centered at the origin. The 
measure \x has necessarily a scaling property: there exists a > such that 
for any Borel set B G M. d and c > 0, fJ.(cB) = c~ a fi(B). We say that 
the probability law F has regularly varying tails with the tail exponent a 



(see Resnick (1987), Hult et al. (2005)), and we view F as the law with 
non-truncated power tails. When studying the extent to which the central 
limit theorem behavior is affected by truncation (which is the main point of 
interest to us in the present paper) we will assume that < a < 2. This 
restriction on the tail exponent a is precisely the one that guarantees that 



F is in the domain of attraction of an a-stable law; see e.g. Rvaceva (1962). 



Such a restriction on the values of the tail exponent will not be necessary in 
other parts of the paper. 

For n = 1,2,... (regarded both as the number of observations in the nth 
row of the triangular array and the number of the model) let M n > denote 
the truncation level. The nth row of the triangular array will consist of 



observations X, 



nj, 



1, . . . , n, which we view as generated according to the 
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following mechanism: 

(1.2) X nj := Hjl^HjW < M n ) + -^(M n + ^)l(ll^ll > M n ) , 

\\ H j\\ 

j = 1, . . . , n, n = 1,2,.... Here H\, H2, ■ ■ . are i.i.d. random vectors in M. d 
with the common law F that has regularly varying tails with a tail exponent 
a G (0,2), and R\,R^... are an independent of H\ y Hi,... sequence of 
i.i.d. nonnegative random variables. For each n = 1,2,... we view the 
observations X n j , j = 1, . . . , n as having power tails that are truncated at 
level M n . 

We need to comment, at this point, on the role of the random variables 
R2, ■ ■ .. One should view them as possessing light tails, even exponen- 
tially decaying tails. In many cases taking these random variables to be 
equal to zero with probability 1 is appropriate; in other applications expo- 
nentially fast tapering off of the tails beyond the truncation point has been 
observed (see e.g. Hong et al. ( 2008[ >). The reader will notice that the results 



of this paper hold whenever the tails of the random variables R\, R2, ■ ■ ■ are 
only light enough, not necessarily exponentially light. We have chosen to 
formulate our results in this way in order to increase their generality, even 



though we are thinking of their role in the model (1.2) as representing the 
exponentially fast decaying tails. 

Our approach to addressing the question "to what extent do models with 
truncated power tails retain the characteristic features of power tails?" lies 
in studying the effect of the rate of growth of the truncation level M n on the 



asymptotic properties of the triangular array defined in (1.2). Specifically, 



we introduce the following definition. We will say that the tails in the model 



fll.2D are 

truncated softly if lim n _ s . 00 nP(||i?i|| > M n ) =0, 



(1.3) 

truncated hard if limn-^ nP[ ||i?i|| > M n ) = 00. 

Clearly, an intermediate regime exists as well. Understanding the various 
aspects of an intermediately truncated model is an interesting theoretical 
question, that we largely leave aside in this paper, in order to keep its size 
manageable (see, however, Remark [T] of Section [2]). For practical purposes 
the soft and hard truncation regimes provide the main dichotomy. 

We will use fairly classical techniques in Section [2] below to show that, 
as far as the behavior of the partial sums of the truncated heavy tailed 



model (1.2) is concerned, observations with softly truncated tails behave 
like heavy tailed random variables, while observations with hard truncated 
tails behave like light tailed random variables. It is, however, clear that, in 
practice, the truncation level M n is not observed. Therefore, we set before 
ourselves two tasks in this paper. The first one, is to estimate the tail 
exponent a based on a sample of observations with truncated power tails 
without knowing the truncation level or, even, if the truncation is soft or 
hard. We show how this can be accomplished in Section |3j The second task 
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is to find out whether the tails in the sample are truncated softly or hard. In 
Section |4j where we suggest statistical procedures for testing the hypothesis 
of the soft (correspondingly, hard) truncation regime against the appropriate 
alternative. In Section [5] we apply the statistical techniques of Section [4] to 
two recent data sets related to TCP connections in a large computer network. 
The goal for that section is only to illustrate the statistical tests discussed 
in Section HI 

We finish this section by pointing out that some of the issues related to 
models with power tails that have been "tampered with", have been ad- 
dressed in the literature. The goals, points of view or the ways in which 
the tails are modified were different from what is done in the present work. 
The paper Asmussen and Pihlsgard ( 2005 ) discusses an application of dis- 
tributions with truncated power tails in queuing, and addresses the question 
whether light tailed approximations or heavy approximations work better 
in this situation. On the other hand, a maximum likelihood estimation pro- 
cedure of the tail exponent a in a parametric model of truncated power 



tails (specifically, the truncated Pareto distribution) is given in Aban et al 
( 2006 ) . Estimation of the tail exponent in randomly censored power models 
(where the tails are not so much truncated, as contaminated) is discussed in 



Beirlant et al.| (|2007j) and |Einmahl et al.| (|2008[). Finally , the H ill estimator 
for censored data is discussed in Beirlant and Guillou| (2001). The tech- 
niques for dealing with censored tails are consistent with the earlier work of 



Csorgo et al. (1986). 



2. A Central Limit Theorem for random vectors with 

TRUNCATED POWER TAILS 



Consider the triangular array defined in (1.2), where, as we recall, the 
random vectors H\, H2, ■ ■ ■ have a distribution with regularly varying tails 
with a tail exponent a G (0, 2). This means that these random vectors (or 
thei r law F) a re in the domain of attraction of some a-stable law p on M rf 
(1962)). That is, the partial sums Sn = H% + . . . + H n , 



(see 
n = 



Rvaceva 



1,2, 



That is, 

converge in law, after appropriate centering and scaling, to p. 



Defining the sums of the truncated observations, 



S n : — ^ ^ X n j, n — 1,2, 



we would like to know whether (S n , n = 1, 2, . . .) still converge in law, after 
suitable centering and scaling, to p. If the answer is no, then we would like 
to know what do these sums of random vectors with truncated power tails 
converge to. These questions can be handled by the classical probabilistic 
tools, and the answer turns out to depend exclusively on the truncation 



regime as defined in (1.3). 
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2.1. Soft truncation regime: truncated heavy tails are still heavy. 

We start with the situation where the truncation level M n grows sufficiently 



fast with the sample size, so that the truncated power tails model (1.2) is 



in the soft truncation regime. Theorem 2.1 below shows that, in this case, 
the partial sums of the random vectors with truncated heavy tails converge, 
when properly centered and scaled, to the same a-stable limit as without 
truncation. 

Let (c n ) and (b n ) denote, respectively, some centering and scaling se- 
quences for the non-truncated random vectors (Hj), that is, 

n 

(2.1) b- l S^ - Cn = 6 -i Y^H 3 -c n ^p 



j'=i 



as n 



oo. 



Theorem 2.1. In the soft truncation regime we have 

(2.2) b^Sn-Cn^p. 



Proof. By (2.1) it is enough to show that 



0. 



However, for any e > 0, 





n 






S n - Hj 






3=1 





< nPiWHA > M n 



and the claim follows. 



□ 



2.2. Hard Truncation regime: truncated heavy tails are no longer 
heavy. Now we consider the situation where the truncation level M n grows 
relatively slowly with the sample size, and that the truncated power tails 
model ( 1.2 ) is in the hard truncation regime. As we will see, in this case the 



partial sums of the random vectors with truncated heavy tails are no longer 
asymptotically a-stable but, rather, converge in law, after suitable centering 
and scaling, to a Gaussian limit. Therefore, at least from the point of view 
of the behavior of partial sums, a model with power tails that have been 
truncated hard does not behave anymore as a heavy tailed model. 

We start with some preliminaries. Recall that, since the limiting law p in 



(2.1) is a-stable, the Levy-Khinchine formula for its characteristic function 



has the form 
(2.3) 



p{6) = exp 



i(0 l7 > 



+ 



,s)l{x < l)}oT (1+a) dx) T{ds) 
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for 9 G M. d , where 7 G M. d , and T is a finite me asure on the unit sphere in 



S:={xe 



1}, see Theorem 6.15 in Araujo and Gine (1980). The 



measure V is often referred to as spectral measure of the law p; see Theorem 
2.3.1 in Samorodnitsky and Taqqu (1994). 

Theorem 2.2. Assume that ER 2 < 00, and let 

B n := [nMlP{\\H x \\ >M n )] 1/2 , n = l,2,... . 
Then in the hard truncation regime we have 

(2.4) B-\S n -ES n )=>ri, 

where n is a centered Gaussian law on M. d whose covariance matrix has the 
entries 

(2.5) -— / SiSjf(ds), i,j = 1, . . . ,d, 

— ct Js 

where T(-) := T(-)/T(S) is the normalized spectral measure of p. 
We start with a lemma. 



Lemma 2.1. For every continuous function f : S 



lim nB n 

n— >oo 



Mr, 



S JO 



f(s)r 2 P \\Hi\\ G dr, 



Hi 



G ds 



a 



" Js 



f(s)f(ds). 



Proof. Assumption (2.1) means that 



(2.6) 



P (\\ Hl \\ > r, ^ € 
PtilHA > r) 



weakly on S; see e.g. Corollary 6.20 (b) of Araujo and Gine (1980). There- 
fore, 



M n 



S JO 



f{s)r 2 P ||#i|| e dr, 



Hi 



M„ 



H 



Hi 



€ ds 



2y (J ZOO*' [\\ H i\\ > V, jngjr G da ) ] d,, 
-Ml j f(s)P\\\H 1 \\> M n , j^eds 



f(s)T{ds) / 2yP(||ffi|| >y)dy-M2p(||F 1 || > M„ 
js 1 Uo 



1 Af*P(p2i|| > M n ) = n~ l B 2 n / /(s)r(cfa) 



as n —¥ 00, where the second asymptotic equivalence follows from the Kara- 
mata theorem (see e.g. Resnick (1987)). □ 
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Proof of Theorem \2.£\ By the Cramer- Wold device it suffices to show that 
for every 9 in M. d , 

B- l ((6, S n ) - E(9, S n )) =► N (o, ^— J {9, s) 2 f(ds)^ . 

To this end we will use the Central Limit Theorem for triangular arrays 



under the Lindeberg condition; see e.g. Theorem 2.4, page 345 in |Gut 
(2005]). We need to prove that 

(2.7) lim -^Var((0,X nl >) = [ (9,s) 2 t(ds) 

n->oo Bfi 2 — a Js 

and that for every e > 0, 
(2.8) 

-^E(\(6,X nl ) - E((6,X nl ))\ 2 l(\(9,X nl ) -E((6,X nl ))\ > eB n 



as n — y oo. In order to prove (2.7), we will show that 

(2.9) lim ^E(((9,X nl )) 2 ) = / ((0, s» 2 f 

n->oo B* ' 2 — a J s 



while 



nV2. 



(2.10) lim — -|£((fl,X n i))| =0. 



The former claim follows easily from Lemma 2.1 and the weak convergence 
(2.6) by writing 

E(((9,X nl )) 2 ) = e(({9,H 1 )) 2 1[\\H 1 \\ <M n 

~ n_1 ^2^ /^'^W+N))^ ( (< w a i(l|gl11 > M " } 

~„-i^_?L_ /" ((0, s )) 2 f(d S ) + M n 2 P(|| J ff 1 ||>Af n ) / «M) 2 f(<fc) 
l ~ a Js Js 

= ^^(^ + l) J{(e,s)ft(ds) = n~ l B« 1 ^- J{{9,s)ft(ds). 
For ( |2.10[ ) we write 

< I^H^dllfillldli?!!! <M n )) +M n P(\\H 1 \\ >M n ) 

Since 

M^dl^xH > M n ) < M n (P(||F 1 || > Af n )) 1/2 = n- 1 / 2 ^ , 
the claim (2.10) will follow once we check that 
(2.11) lim n^B^E [||fli||l(||.Hi|| <M n )] = 0. 



8 



A. CHAKRABARTY AND G. SAMORODNITSKY 



We give separate arguments for the cases a < 1 and a > 1. 

Case 1 (a < 1): Letting C be a positive constant whose value may change 

from line to line, by the Karamata theorem, 

iH.f/HiWH^l < M n )]) 2/3 



<M n ) 



< 



(E 



~ CM n (P(||fri|| >M„)) 2/3 

= Cn- 1 /2 Sft (p(||^ 1 || >M„)) 1/6 

< n~ 1/2 £ n - 



Case 2 (1 < a < 2): Here (2.11) follows trivially from the fact that 
E [||J3i||l(||fli[| < M n )\ has a finit e limi t, while B n > n 1 / 2 as a < 2. 

We have now proved (2.7). By (2.10), the remaining condition (2.8) will 
follow once we check that for every e > 0, 



n 

Bl 



E 



. X n i) 



. X n i) > eB n 



0. 



This is, however, an immediate consequence of the fact that the hard trun- 
cation implies that B n 3> M n as n — > oo. □ 

Remark 1. We briefly address the behavior of the partial sums of the 
random vectors with truncated heavy tails in the intermediate regime 



(2.12) 



lim nP(\\H\\ > M n ) = 5 £ (0,oo) 



It turns out that, in this case, one can use the same centering and scaling 
sequences {c n } and {b n } as in the non-truncated case (2.1) (or in the soft 
truncation regime (2.2)), but the limit will be different. In fact, 

(2.13) 



b„ S; 



n 



PS 



where ps is an infinitely divisible law on which is obtained by a certain 

;he a- 

ps(0) = exp 



truncation of the jumps of the a-stable law p in (2.3). Specifically, 
(2.14) 



+ 



Ss\Sq 



| e «(»,«> -l-ix(9,s)l(x< 1)}^-( 1+Q ) dx 



for £R d , where 

- a l(x <l)dx [ sT(ds). 
Js 



is = 7 , 

/5-V«(a-ir(5)) 1 / 
We sketch the argument. Write 

n 

(2.15) b-'Sn-c 



bn l Y. H M\\ H l\\< M n 
3=1 
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+K 1 M 1 



wrMim >M n ) +b- i j2 jjhRMimi >M n ). 
j= i ii i ii j= i ii j i 



It is easy to check that the last term in the right hand side of ( 2.15 ) converges 
to zerc 

(2.16) 



to zero in probability. Since (2.12) implies that 

Mn 

as n — > oo, Theorem 5.9, p. 129, of Araujo and Gine (1980) implies that the 



first term in the right hand side of (2.15) has a weak limit whose character- 
istic function is given by 



+ 



S \J0 



5-i/«( a -ir(s))V° 



exp 

Jx(8,s) _ 



l-ix(6,s)l(x < l)}x" (1+a) dx\v(ds) 



Finally, it follows from (2.16) that the second term in the right hand side of 



(2.15) is asymptotically equivalent to 



(T^oT^S)) 1 /^ 



-l{\\Hj\\ > M n ) , 



and, by (2.6) and (2.12), the sum above converges weakly to the law of the 
Poisson sum Y2j=i where Y\,Y2, . . . are i.i.d. S- valued random variables 
with the common law T, and ./V is an independent of them Poisson random 
variable with mean 5. Since the weak limits of the first and the second terms 



(2.13). 



in the right hand side of (2.15) are easily seen to be independent, this shows 



3. Hill estimator for random variables with truncated power 

TAILS 

Estimating the tail exponent a is one of the main statistical issues one 
faces when working with data for which a model with power tails is con- 
templated. This is a difficult statistical problem because one attempts to 
estimate a parameter governing the tail behavior in an otherwise nonpara- 
metric model. By necessity, any estimator one uses has to be based on a 
vanishing fraction of the available data. The situation is even trickier when 
one tries to estimate the tail exponent in a sample of observations with 
truncated power tails. This is the task we address in this section. 

The formal setup in this section is as follows. We are given a sam- 
ple X\, . . . , X n of one-dimensional nonnegative observations from the 



model with truncated power tails, i.e. (1.2). We emphasize a slight change 
in notation from (1.2): whereas the latter used the notation X n i, 



to emphasize the triangular array nature of the model, in a statistical pro- 
cedure, when a single sample (i.e., a particular row of the triangular array) 
is given, the notation Xi,..., X n is more natural. The discussion in Section 
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[2] makes it intuitive that estimating the tail exponent a should be easier if 
the tails are truncated softly, than in the case when the tails are truncated 
hard. This is, indeed, the case. However, in this section we are interested in 
finding a procedure that permits consistent estimation of the tail exponent 
a regardless of the truncation regime; this is especially important because 
the truncation regime is never known (see, however, Section [4] below) . Fur- 
thermore, in this section we do not restrict the values of the tails exponent 
to the interval (0, 2). That is, a can take any positive value. 

A number of estimators of the tail exponent of distributions with non- 
truncated power tails have been suggested; a thorough discussion can be 



found in Chapter 4 of de Haan and Ferreira ( 2006 ) . One of the best known 



and widely used estimators is the Hill estimator introduced by Hill ( 1975 ) 
Given a sample X%, 



,X n . the Hill statistic is defined by 



(3.1) 



h 



n,k 



5> 



X 



X 



(k) 



where Xm > X( 2 ) > • • • > -^M are the or der statistics from the sample 
X\, . . . , X n , and k = 1, . . . , n is a user-determined parameter, the number of 
the upper order statistics to use in the estimator. The consistency result for 
the Hill estimator says that, if X±, . . . , X n are i.i.d. with regularly varying 
right tail with exponent a > 0, and k = k n — > 00, k n /n — > as n — > 00, then 
h n ,k n — > 1/a in probability as n — > 00; see e.g. Theorem 3.2.2 in de Haan 



and Ferreira (2006). 



In spite of the simplicity of the statement of the consistency of the Hill 
estimator, selecting the number k of the upper order statistics for a given 
sample with nontruncated power tails remains a daunting problem; see e.g. 
pp. 192-193 in Embrechts et al. (1997). In the main result of this section, 



Theorem 3.1 below, we will see that one has to be particularly careful when 
using the Hill estimator on a sample with truncated power tails. Nonetheless, 
a consistent estimator can still be obtained. 

Notice that the next theorem does not impose any conditions on the 



random variables R\,R2, ■ ■ ■ in the model (1.2). 



Theorem 3.1. Suppose that the number k n of the upper order statistics 
satisfies 



(3.2) 

Then h n & 



nP(Hi > M n ) + K k n < n . 
1/a in probability as n — > 00. 



Note that Theorem 3.1 says that, in the soft truncation regime, the Hill 
estimator is consistent under the same assumption, k n /n — > 0, as in the 
nontruncated case. 



Proof. For simplicity, we write k instead of k n . An inspection of the proof 



of consistency of the Hill estimator in the nontruncated case in e.g. Resnick 
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(2007) shows that the result will follow once we check that, under the con- 
sul 



ditions of the theorem, 
(3.3) 



n 
k' 



P 



b(n/k) 



e 



/*(• 



vaguely in (0,oo], where fi is a measure on (0, oo] defined by 
//((a:, oo]) = x~ a for all x > , 

and 

b(u) = inf{x > : P{H X > x) < u' 1 }, u>0. 
Note that (b(n)) is no longer necessarily a sequence satisfying (2.1). In 



fact, we will use this notation several times in the sequel to denote other 
quantile-type functions associated with the random variable H%. 
By the hypothesis, 



77 

lim 7 P( J ff 1 > M n ) 

n— >oo K 







and, hence, b(n/k) <C M n as n — )• oo. Therefore, for any x > 0, for n large 
enough, 



b(n/k) 



> x 



P(#! > xb{n/k)) 
k 



n 



where the last line follows from the hypothesis k <C n and regular variation 
of the tail of H\. This shows (3.3). □ 



Since the truncation level M n is not known, it is desirable to have a 
sample-based way of deciding on the number of upper order statistics to use 



in the Hill estimator. A natural (in view of the condition (3.2)) choice is to 
use a random number of upper order statistics given by 



(3.4) 



n 



(n ^ 1 
3=1 



j > 7 max X{ 
i=l,... ,n 



where 7 and /3 are user-specified parameters taking values in (0, 1) and [•] 
denotes the integer part. The next two theorems show that this choice of 
the number of upper order statistics leads to a consistent estimator of the 
reciprocal of the tail exponent. The parameters f3 and 7 can and should be 
chosen by the user, and the consistency of the estimator depends neither on 
their choice nor on, for instance, second order regular variation of H\. In 
practice, one should, probably, use some version of the so-called Hill plot 



(see e.g. Embrechts et al. (1997)), by calculating the value of the estimator 



for a range of (3 and 7. We leave this issue for future work. 

We start with proving consistency of the Hill estimator with the random 
choice of the number of upper order statistics in the hard truncation regime. 
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Here the consistency requires that the tail of the random variable Ri is 
sufficiently light, relatively to the truncation level. 

Theorem 3.2. In the hard truncation regime assume, additionally, that 
(3.5) P(fli > M n )P{R 1 > eM n ) = o(l/n) for all e > 0. 



If k n is chosen as in (3.4), then h n j c — >■ 1/a in probability as n — > oo. 
Proof. We start by showing that, as n — > oo, 

K P 



(3.6) 
where 



k n 



1 



kn '■- 



nP(Hx > 1 M n f 



Clearly, all that needs to be shown is 

£?=l 1(X, > 7 M n 



(3.7) 



nP{H x > 7 M n ) 



1, 



where M n := maxi<i< n Xj. By (|3_5j), 

Mn P 



(3.8) 



Further, because of the hard truncation, 

Ei=i 1 (i : ^ > 



(3.9) 



nP(H > 6 2 M n ) 



for any < B\ < 1 and 6> 2 > 0. 

Fix < e < 1. Choose first < n < 1 be such that (1 - r])~ a < 1 + e. 
Writing 



£" 1 (X, > 7 M r 



P 

nP(#i > 7 M f 
< P[(l-rj)M n > M n ] + P 



> 1 + e 

'E-=il(^'>7(l-r/)M„ 



nP(ffi > 7 M„ 



> 1 + e 



and using (3.8) and (3.9), we see that 



P 



nP(Fi > 7 M n ) 



> 1 + e 



0. 
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Next we choose r\ > such that 7(1 +77) < 1 and (1 + r/) a > 1 — e. Writing 

E"=llfc>7Mn' 



p 



nP{H x > 1 M n ) 



< 1-e 



< P[(l + V )M n < M n ] + P 



E-=il(^>7(l + r?)M„ 
nP(Hi > 7 M n ) 



< 1-e 



and appealing, once again, to ( |3.8| ) and ( |3.9| ), we obtain 

— >0, 



P 



nP(Pi > 7 M n ) < 1 



which establishes (|3. 7|) , and, hence, also (|3.6 ) 

ces to sh 

-1 ^'n 

£5> 



In view of (3.6), it suffices to show that 

p l 



kn f^i x {k n ) 



a 



Fix e > 0. We choose 0<r/<l/2so that a log < |, and write 



P 



< P 



i=i 



2 ^ra 



^71 



5> 

8=1 



-P 





1 


> e 




Q 




1 


e 

>3 




a 









+ p 



log 



X 



3 



([*»(!-»?)]) 



It follows from Theorem 13, II that 



P 



(0 



^ X (fcn+1) a 



> 



Since k n 3> nP(H\ > M n ), we see that 

*([Mi+u)]) p , ^ 1 + 7 A _1/a 



([*»(!-»»)]) 



Therefore, by the choice of 77, 



P 



log 



([*n(l+fl)]) 



X, 



> 



([kn(l-v)]) 



0. 



In conjunction with (3.6), this completes the argument. 



□ 
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Next, we show consistency of the Hill estimator using the random number 



(3.4) of upper order statistics in the soft truncation regime. Note that no 
assumption on the tail of Ri is necessary in this case. 



Theorem 3.3. In the soft truncation regime, if k n is chosen as in (3.4), 
then h n ^ — > 1/a in probability as n — >• oo. 

Proof. Let {h{k, n)) denote the Hill sequence based on the random variables 
Hi, ... , H n , i.e. 

h(k,n) : ' V log " 



where 1 < k < n and Hm ^ ... > are the order statistics of 

Hi,... ,H n . Let f3 be as in (3.4). It is well known that the random step 
function 

fhdn 1 -^],^ :t> l) 

converges in probability in D[l, oo), equipped with the topology of uniform 
convergence on bounded intervals, to the constant deterministic function 
c(t) = 1/a for t > 1; see Resnick (2007), page 89. Since 

P (fr([n 1-/J i],n) / h{[n l ~ p t],n) for some t > l) < nP(H > M n ) — >0, 

it follows that 

(3.10) ^([n 1_ ^t],n) : t > lj A (c(t) : t > l) 

in D[l,oo) as well. By Proposition 3.21 (page 154) in |Resnick (1987) and 
soft truncation, we known that 

n oo 

N n := Tl{Xj > 7 max X t ) => AT := V 1 (rT 1 /" > 7 r- 1/Q 
' — * 1=1,... ,n ' — ' V J 

3=1 3=1 

where (r^- : j > 1) is the sequence of the arrival times of the unit rate Poisson 



process on (0,oo). Using (3.10) and Theorem 4.4 (page 27) in Billingsley 
( fl968| ) we conclude that 



(hitf-HU), n£) ==> (± Nf> 



in D[l, oo) x N' 3 , where N' 3 := {1, 2°, 3^, . . .}. Since the evaluation map from 
D[l, oo) x to 1 defined by {x, a) h-> x(a) is continuous, an appeal to the 
continuous mapping theorem finishes the proof. □ 

4. Testing for soft and hard truncation 

The first two sections of this paper provide, among other things, evidence 
that, in certain important respects, random variables with truncated heavy 
tails retain "most of the tail heaviness" if the truncation is soft, but loose 
"much of the tail heaviness" if the truncation is hard. Since the truncation 
level is not observed, how does one decide if the tails of observed data have 
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been truncated softly or hard? In this section we construct statistical tests 
for testing each of the two hypothesis against the corresponding alterna- 
tive. As in Section [3] we restrict ourselves to the case of one-dimensional 
observations and the tail exponent a can take any positive value. 

Suppose that we are given a sample X\ , . . . , X n of one-dimensional ob- 
servations from the model (1.2). As in Section [3j we do not use here the 



triangular array notation. Neither the precise value of the tail exponent nor 



the exact distribution of the random variables (R n ) in (1.2) are assumed 
to be known. However, we will assume that an upper bound on the tail 
exponent a is known. 

This section is split into three subsections, describing, correspondingly, 
testing the hypothesis of soft truncation, testing the hypothesis of hard 
truncation, and testing a slightly stronger version of the latter. 

4.1. Testing the hypothesis of soft truncation. We consider the fol- 
lowing problem of testing a null hypothesis against a simple alternative: 



(4.1) 



H : P(\Hx\ > M) <C n (soft truncation) 



H^t ■ P{\Hi\ > M) > rT x (hard truncation) 
We assume the tail exponent a satisfies 
(4.2) a<yl<oo, 

i.e. an upper bound on the tail exponent is available. As a test statistic we 
will use 

zr=i \^\ A 



(4.3) Z n (A) :-- 



maxi<j< r 



The following proposition describes the asymptotic distribution of Z n (A) 
under the null hypothesis and under the alternative. 

Proposition 4.1. (i) Under the hypothesis Hq of soft truncation, 

oo 

(4.4) z n (A)^rf/ a J2 T 'j 



3=1 



where (Tj, j > 1) are the arrival times of a unit rate Poisson process on 
(0,oo). 

(ii) Assume that ERf < oo. Then under the hypothesis H a \t of hard 
truncation, Z n (A) — > oo. 

Proof. For part (i), we define 

b n = infja; > : P{\Hi\ A > x) < ra" 1 }, n = 1,2,... . 

Note that, for any x > 0, 

nP^I-Xil^ > x) ~ nP(b- l \H!\ A > x) -> x~ a 



16 



A. CHAKRABARTY AND G. SAMORODNITSKY 



as n — > oo. It follows from Proposition 3.21 (page 154) in Resnick (1987) 
that we have the following weak convergence of a sequence of point processes 
on (0, oo]: 

n oo 

(4.5) N n :=J2 d bnW N := J2 S ft^ 

j=i j=i J 

as n — > oo. Here 5 a is a point mass at a, and the weak convergence takes 
place in the space of Radon point measures on (0, oo] endowed with the 
topology of vague convergence; see Section 3.4 in Resnick (1987). We would 
like to use the continuous mapping theorem to deduce (4.4) from (4.5), but 
a preliminary truncation step is necessary. 
For e > we define 



Z n (A;e) :-. 



maxi<j< n 



Notice that Z n (A; e) = h(N n ), where for a Radon point measure r\ = ^ - S rj 
on (0, oo], 

maxj rj 

It is standard (and easy) to check that h is continuous with probability 1 
at the Poisson random measure N in (4.5), so by the continuous mapping 
theorem, 



Z n (A;e) => r*' al £ i TJ A ' a l(TJ A ' a > e) . 

3=1 

Therefore, the convergence (4.4) will follow once we check that for every 
5 > 0, 



(4.6) 



lim lira sup P(Z n ( A) - Z n (A; e) > 5) = . 

e ~ ^0 n— »oo 



To this end, notice that, for any < < 1 we can select r > so small that 
P(maxi<j< n \Xi\ A < rbn) < for all n large enough. Then, for all n large 
enough, 

P{Z n (A) - Z n {A; e)>5)<0 + S^E (r^b' 1 £ \ x i\ A ^-{K l \Xi\ A < 

= + r 1 r- 1 n6- 1 J B('|Xi| A l(&- 1 |Xi| A < e) 
= + S^T^nb^E^H^l^-^H^ < e) 
-0 + rV" 1 ^; 1 ((1 - a/A)- 1 (eb n )P[\H 1 \ A > eb n ) 

-0 + S-^nb-^l - a/A)- l {eb n ){e~ a l A n- 1 ) 
= + tV-^l - a/A)- 1 e 1 - a l A . 
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where the second equality holds because of soft truncation, and the first 
asymptotic equivalence follows from the Karamata theorem. Since A > a, 
we obtain ( |4.6[ ) by first letting e — > and then 9 — > 0. This completes the 
proof of part (i). 

For part (ii), we start with observing that 

(A7) Si ' X '| A > l^! A l(^n/2 < \Hi\ < M n ) 

1 • ) nM A P{\Hx\ > M n ) ~ nM^PdHjl > M n ) 

>(M , ^1=1^/2 <m<M n ) 

* {MJ2) nM A P(\ Hl \>M n ) 2 (2 " 1} 

in probability. On the other hand, for some constant c > 0, by the assump- 
tion ER A < oo, 

max \Xi\ A <c(M A + max Rf) = cM A + o(l)n 

l<i<n V n l<j<n ' n 

a.s. as n — > oo. Since the truncation is hard, and A > a, we see that 

(A H) max i=1 „„, n \Xj\ A 

nM^P(\H 1 \>M n ) 



a.s. as n — > oo as well. The claim of part (ii) follows from (4.7) and (4.8). □ 



Based on Proposition 4.1, we suggest the following test for the problem 



([41]). 

(4.9) reject Hq at significance level p £ (0, 1) if Z n (A) > c p (a/A) , 
with c p (9) such that P(Z(9) > c p {9)) = p, where for < 6 < 1, 

oo 

(4.10) Z{9) =T 1 1 /6 Y, T J 1/e ■ 

j=i 

The random variable Z(9) does not seem to have one of the standard 
distributions, and we are not aware of any previous studies of the distribu- 
tion of Z(9). The following proposition lists some of the properties of this 
distribution. 

Proposition 4.2. The random variable Z{9) is an infinitely divisible ran- 
dom variable. It has a density with respect to the Lebesgue measure, and the 
Laplace transform 

(4.11) Ee^ m = (l + 7 e 7 £ e^ x x~ e dx^j , 
7 > 7o, where 70 < is the number satisfying 

1 + 7o e 70 f e-^ x x- e dx = . 
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,-1/ 



Proof. For 5 > let 

oo 
3=1 

Then is an infinitely divisible random variable with the Laplace trans- 
form 



.Be 



-fWs 



exp 



'1 



-72/ 



)0y 



-(1+9) 



dy 



for all 7 G M because the Levy measure of Wg has a compact support; see 
Rosihski| ( fl990| ) and |Sato| (l999| >. Since 

Z(0) = 1 + T l / W T 

where T is a standard exponential random variable independent of (Tj : j > 
l), it follows that 



(4.12) 



exp 



-r^ex-V+e) dx \ dt 









-7 r / , 

e ' / exp < — t 


e 7 + 7 / e 7:c x 61 


| dt 




JO 





via integration by parts. Since the exponent under the integral is positive 
if and only if 7 > 70, we obtain (4.11). Additionally, it follows from (4.12) 
that 

(4.13) Z(9) = 1 + Y(T), 

where (Y(t), t > 0) is a subordinator satisfying 

Ee~^ Y ® 



(4.14) 



exp 



(l-e- rx )6x-( 1+e Ux\ , t>0, 



independent of T. Since a Levy process stopped at an independent infinitely 
divisible random time is, obviously, infinitely divisible, so is Z{&). Further- 
more, the characteristic function of Y(t) is integrable of the real line for 
every t > 0, so each Y(t) has a density, and then the same is true for any 
mixture of (Y(t)). Therefore, Z(9) has a density. □ 



Even though we know, by Proposition 4.2, that the random variable Z{9) 
has a density, at present we do not know ways to compute this density. 
One possibility to estimate the critical values c p (a/A) to perform the test 
(4.9), is as follows. For values of a not too close to the upper bound A (or, 
equivalently, for the values of 6 not too close to 1), it is possible to estimate 
the critical values by the Monte-Carlo method, by truncating the infinite 
series at a sufficiently large finite number of terms. Using ./V = 10 5 number 
of terms in the series and generating the (truncated) random variable 10 5 
times, we have estimated the following quantiles, for a range of values 9. 
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9 

P ^\ 


0.5 


0.6 


0.7 


.05 


4.3 


5.8 


8.2 


.025 


5.1 


6.9 


9.8 


.01 


6.2 


8.4 


12.1 



For 9 closer to 1, the rate of convergence of the truncated sum J2jLi Fj 
as N — > oo is very slow, and in order to obtain upper bounds on the quantiles 



,-1/ 



of the random variable Z(9) we used Proposition 4.2 as described below. 
Such upper bounds lead to conservative versions of the test (4.9). We use 
the exponential Markov inequality: for < r < 



P{Z(9) >z)< e~ rz Ee rZ = e~ r 
and estimate the integral from above by 



1 



re 



-7o, 



Jo 



e rx x~ 



dx 



-i 



c x- e dx < e T ' k 



k' 



)-i 



i=2 v 



k > 1. Using r = .05 and k = 10 we computed numbers c p (9) satisfying 

P{Z(9)>c p (9)) <p. 
These numbers c p (9) are reported in the following table. 



\. 9 
P ^\ 


0.8 


0.9 


0.95 


.05 


65.43 


73.12 


127.37 


.025 


79.29 


86.98 


141.23 


.01 


97.62 


105.31 


159.56 



Since we are only assuming that the tail exponent a has a known upper 



bound as in (4.2), but the exact value of a may be unknown, a possible way 



to obtain a conservative estimate of the critical value c p (a/A) in (4.10) is 



to choose a number At > A and use the statistic Z n (A\) instead of Z n (A) 



in (4.3). By Proposition 4.1, under the null hypothesis, the test statistic 
converges weakly to Z{a,/A\), which is stochastically smaller than Z(A/A\), 



and we obtain a conservative test by modifying (4.9) as follows: 

(4.15) reject Hq at significance level p £ (0, 1) if Z n (Ai) > c p {A/A\) . 

4.2. Testing the hypothesis of hard truncation. In this subsection we 
consider the following problem of testing a null hypothesis against a simple 
alternative: 



(4.16) 



#alt 



P( 



> M) » n -1 

> M ) <C rr 1 



(hard truncation) 
(soft truncation) 
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We still assume that an upper bound (4.2) on the tail exponent is known. 
For a test statistic in this case we choose a number 7 £ (0, 1) and define 

Et1(-i)^f /2> ^ 2 



(4.17) Z n (A; 7 ) 



2^=[ 7 n]+l lAjl 



Here = |a| b sign(a) for real a, b is the signed power. The asymptotic 
distribution of Z n {A; 7) under the null hypothesis and under the alternative 
in (4.16) is described in Proposition |4.3| below. Recall the standard notation 
of S a (a, (3, fi) for (the distribution of) an a-stable random variable with the 
scale a, skewness /3 and location //; see Samorodnitsky and Taqqu (1994). 
For a symmetric a-stable random variable, f3 = ji = 0. For a positive strictly 
a-stable random variable with < a < 1, one has f3 = 1 and [i = 0. Finally, 
for < a < 2, let 



a, 



(r(l -a)cos(vra/2))- 1 ifa/1, 
2/vr ifa = l, 



Proposition 4.3. (i) Assume that ER\ A < 00. Then under the hypothesis 
Hq of hard truncation, 

(4.18) Z n (A; 7 )^C 1 ( 7 )x?, 

where 6*1(7) = 27/(1—7), andxf is the standard chi-square random variable 
with one degree of freedom. 

(ii) Under the hypothesis H & \ t of soft truncation, 

(4.19) Zn (A; 1 )^C 2 (A; 1 )^, 
where 

^7)=(^-^) A/ °, 

V 1 ~ 7 ^2a/A/ 

and S± and S% are independent random variables, such that Si is a symmet- 
ric la I A-stable random variable with unit scale, and S2 is a positive strictly 
a /A-stable random variable with unit scale. 

Proof. The claim of part (i) will follow from the following two statements. 

py^i] 1/2 

(4 20) 1 T(-iyx {A/2) ( -^-) MO 1) 

and 



it 



f 4 on t v \y\ a > A 0--y) 

^■ ZL) nM£P(\Hi\>M n ) . f^J Jl A-a 

j=[ 7 n] + l 
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1 



in probability. We prove (|4.21|) first, and it is enough to show that 
(4.22) 
and 



nMfiP(\H 1 \ > M n 



e[ £ l^il 

, jf=[ 7 n]+l 



A(l- 7 ) 
A- a 



(4.23) 



{nM£P(\H x \ > M n )) 
Note that by the Karamata theorem, 

4 t \x,\ 

W=[ 7 n]+1 



Var[ £ \X 3 \ 
j=[jn]+l 



0. 



(l-^nE^X^) 



(1 - 7 )n P(|#i| A l(|Pi| < M n )) + E(M n + i? 1 ) A P(| J ff 1 | > M n ) 

Q: 



(l-7)n 



A- a 



M^P{\H X \ > M n ) + M£P(\H X \ > M n 



n 



M^P(|ffi| > M„)) 



-4(1-7) 
A-a ' 



proving (4.22). A similar calculation gives us 



Var[ £ 

U=[7"]+l 



;i - 7 )nVar(|Xi 



< nE^X^) ~ (nM 2A P(|#i| > M n )) 



2A 



2A-a ' 



and (4.23) follows because the truncation is hard. Therefore, we have estab- 
lished (14.21). 



In order to prove (4.20), note that the triangular array 



X nj := H\ AI2) 1{\H 3 \ A I 2 < M^) + ^-(M^+R AI2 )l{\H 3 \ A " > M^) , 



7 A/2, 



j = 1, . . . , n, n = 1, 2, . . ., satisfies the assumptions of Theorem 2.2 (with a 
replaced by 2a/ A), and, therefore, 



[nM£P{\H x \ > M n )) 



in n 
T/2 \ ^2^nj-E (J2 Xnj 



The random variables (X 



{A/2), 



J=l 



2A 
A - a 



1/2 



#(0,1). 



form a somewhat different triangular array, 



namely 

X%' 2> = tf< A/2) l(|tf#/ 2 < M n A / 2 ) + ^(M n+J R,) A/2 l(l^| A/2 > M n ^ 2 ) , 



Hi 
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j = 1, . . . , n, n = 1,2,..., but an inspection of the proof of Theorem 2.2 



shows that the argument applies equally well to the latter triangular array, 
so that 



1 



(nM A P(\H 1 \> M n )) 



2A 

A- a 



1/2 



N(0,1) 



In particular, (extending the length of the rows of the triangular array) we 
see that 



1 



2n 



{nM£P{\H x \ >M n )) 1/ 

AA ^ 1 - 



I x {A/2) - r (A/2) 

1/2 | «J nj 

j=l j=n+l 

N(0,1). 



A- a 



Replacing n with [727/2], we obtain (4.20) and, hence, finish the proof of 
part (i). 

For part (ii), we define 

b n = M{x > : P{\Hi\ A/2 >x)< n" 1 }, n = 1,2, ... . 

Then for some centering sequence (c n ) we have 



<A/2> 



with y having a Sig /^a, f3, fj,) distribution with a 2a l A = (C 2q! m) 1 and 
some see |Feller (1971). Bec ause of the soft truncation, the triangular 



2.1 



array [X^j ) satisfies Theorem 



and so 



nj 



Y 



J =1 



with the same Y. Extending the rows of the triangular array gives us 



2n 



\ j= l j=n +l I \^2a/Aj 



(A/2) 



2 \ A /( 2a ) 



Si, 



"2a/ A, 



where Si is a symmetric 2a/j4-stable random variable with unit scale. Re- 
placing n with [717/2] we obtain 

M x _ \A/{2a) 



(4.24) 



E(- 1 ) JX . 

i=i 



<A/2> 



Cia/Aj 



Si 
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Next, we also have 



a 



a/ A 



A I a 



s 2 . 



where 52 is a positive strictly a/A-st&ble random variable with unit scale; 
see once again Feller (1971). As before, because of the soft truncation, 
Theorem 2.1 applies, and we obtain 

A/a 



K 2 ~^2 \Xnj\ A => 
3=1 

Replacing n with (1 — 7)71, shows that 



1 



a 



So. 



(4.25) 



K 2 E 

j=[ 7 n]+l 



a/A 



l- 7 



a 



a/ A 



A/a 



s 2 . 



Since the numerator and the denominator of the statistic Z n (A;j) in (4.17) 
are independent, the claim of part (ii) of the proposition follows from (4.24) 
and (IO§|). □ 



Interestingly, the asymptotic distribution of the test statistic Z n (A; , y), 
under the null hypothesis, does not depend on the choice of the parameter 
A (as long as it an upper bound on the tail exponent a). Furthermore, 
under the null hypothesis this asymptotic distribution of the test statistic 
is light-tailed (e.g. some exponential moments are finite). On the other 
hand, the asymptotic distribution of the test statistic under the alternative 
is, clearly, heavy tailed, as even the second moment is infinite. Therefore, a 
reasonable test will reject the null hypothesis in favor of the alternative if 
the test statistic is too large. That is, we suggest the following test for the 



problem (4.16) 
(4.26) 



2 7 



reject Hq at significance level p £ (0, 1) if Z n (A; 7) > ^ p , 

with c p such that P{x\ > c p ) = p. 

4.3. Testing a stronger version of the hypothesis of hard trunca- 
tion. The test statistics Z n (A;j) we used in the previous subsection for 
the problem (4.16) has a nondegenerate asymptotic distribution under both 
the null hypothesis and the alternative. This may restrict the power of the 
resulting test. In order to obtain a more powerful test we strengthen the 
null hypothesis. Specifically, in this subsection we consider the following 
problem of testing a null hypothesis against a simple alternative: 

H : n l ~ e P{\H l \> M) » 1 
# a it : nP(\H 1 \> M) « 1 



(4.27) 



where e is a fixed number in (0, 1). 
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For this problem one can use the same test statistic Z n (A) defined in (4.3) 



as we used for the problem (4.1 ) of testing the hypothesis of soft truncation. 



Proposition 4.1 tells us that this test statistic diverges in probability to in- 
finity under the hypothesis of hard truncation. The strengthened hypothesis 
of hard truncation in ( |4.27 ) allows us to quantify how fast this divergence 
takes place. This, in turn, can be used to build a test. The asymptotic 
distribution of Z n {A) under the hypothesis of soft truncation is described 
in Proposition |4.1| The next result provides an asymptotic distributional 
lower bound on the test statistic under the null hypothesis in the problem 



(4.27). As in the previous subsections, we assume that an upper bound (4.2) 



on the tail exponent is known. 

Proposition 4.4. Assume that ER\ A < oo. Then under the strengthened 
hypothesis Hq of hard truncation, 

(4.28) lim inf p( n- t/2 ZJA) > x) > e~ x2 

n— >oo \ / 

for every x > 0. 



Proof. In the notation of the triangular array (1.2), consider the binomial 
random variable N n = J2j=i > M n ). The strengthened hypothesis 



of hard truncation implies that P(N n > n e ) 
on an event of probability increasing to 1, 



1 as n — > oo. Notice that, 



Z n {A) > 



E?=l(A*n + flj) A l(|flj| >M n ) 

max., =Wl (M n + R j ) A l(\H j \ > M n 



> 



E]=l R t 1 (\ H j\> M n) 



max j= i,..., n Rfl(\Hj\ > M n ) ' 
Therefore, for x > 0, using the assumption ER± A < oo, we have 



liminf P[n- e/2 Z n (A) > x) > lim inf P[ max Rf< n - e/2 N n ^x" 1 

n— s>oo \ / n— >co \j=l,...,N n 2 



> lim inf E 

n— voo 



X 



1 {N n > n t 



as required. 



□ 



Proposition 4.4 tells us that under the hypothesis Hq, n~ e l 2 Z n (A) is, 
asymptotically, stochastically larger than the square root of the standard 
exponential random variable (independently of the parameter ^4). Therefore, 
we suggest the following test for the problem (4.27). 
(4.29) 



reject Hq at significance level p S (0, 1) if Z n (A) < |log(l — p) 



V 2 e/2 
n 1 
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0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 



Figure 1 . Think Times - the entire data set 

5. Think Times and Object Sizes data: soft truncation or hard 

truncation? 

In this section we applied the statistical methods of Section [4] to two 
data sets. One data set contains "think times" , or delays (in microseconds) 
between successive request/response exchanges between hosts using a TCP 
connection. The second data set contains the sizes (in bytes) of objects (files, 
HTTP responses, email messages, etc.) transferred on TCP connections. 
Both data sets were acquired by monitoring between 1:30 PM and 2:30 
PM on July 24, 2006, the communication links connecting the site of a large 
commercial enterprize to the Internet. Both data sets exhibit visual evidence 
of heavy tails, and the Hill estimator confirms that (see below). Our goal is 
to check if the data sets show statistical evidence of soft or hard truncation 
of heavy tails. As we will see, in a number of cases we are not able to reject 
either the hypothesis of soft truncation or that of hard truncation. This is an 
indication that there is much room for improving the statistical techniques 
of testing for the type of the truncation of the tails that provides the best 
approximation to data. We will comment more on that issue at the end of 
the section. 

5.1. Think Times. This data sets contains 2.1 x 10 7 observations which 
are plotted on Figure [TJ 

Clearly, the nature of this data set changes over time, and the nature of 
truncation of heavy tails may potentially change as well. In order to study 
this effect we have, somewhat arbitrarily, broken the data set into four pieces, 
with corresponding ranges [0.11 x 10 7 ,0.64 x 10 7 ]; [0.8 x 10 7 ,1.6 x 10 7 ]; 
[1.7 x 10 7 , 1.9 x 10 7 ] and [1.95 x 10 7 ,2.1 x 10 7 ] . The individual pieces are 
plotted on Figure [2] 




The structure of the 4 individual pieces appears to be more stable than 
that of the entire data sets, and we proceed to analyze each piece separately. 



To do that, we first ran the Hill estimator with random k given in (3.4) 
on the first half of each of the 4 pieces. The estimation was conducted 
using j3, 7 = 0.3, 0.4, 0.5, 0.6, 0.7 and conservative upper bounds for a were 
obtained; these are presented in the following table. 



piece 


A 


1 


3.02 


2 


2.30 


3 


0.85 


4 


2.24 



We then proceeded to use the second halves of each piece of the Think Times 
data set to test for soft and hard truncations. 



Testing the hypothesis of soft truncation. The test statistic Z n {A\) of Section 
4.1 was computed for various values of A\ larger than A. The results are 
reported in the following table. 
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A/A, 


piece 1 


piece 2 


piece 3 


piece 4 


0.5 


31.43 


5.81 


154.05 


3.57 


0.6 


51.59 


7.99 


205.37 


4.72 


0.7 


77.39 


10.74 


271.27 


6.11 


0.8 


108.08 


14.20 


361.74 


7.81 


0.9 


142.78 


18.57 


491.31 


9.91 


0.95 


161.38 


21.16 


576.73 


11.13 



Comparing the resulting values of the test statistic with the corresponding 
quantiles (or their upper bounds) of Z(A/A\), it is clear that the null hy- 
pothesis of soft truncation can be rejected for pieces 1 and 3 at the levels 
less that 0.01, while for piece 2 the null hypothesis of soft truncation can be 
rejected at the level 0.025, but not at the level 0.01. For piece 4 we cannot 
reject the null hypothesis of soft truncation even at the level 0.05. These 
findings are rather consistent with visual analysis of the pieces of the data 
set: piece 4 exhibits extremes that do not appear to be seriously truncated 
(one has to keep in mind that the visual analysis can easily be misleading). 



Testing the hypothesis of hard truncation. The test statistic Z n (A; 7) of Sec- 
tion 4.2 was computed for various values of 7. The resulting p-values are 
reported in the following table. 



7 


piece 1 


piece 2 


piece 3 


piece 


0.1 


0.85 


0.72 


0.88 


0.33 


0.2 


0.83 


0.98 


0.38 


0.57 


0.3 


0.97 


0.99 


0.79 


0.68 


0.4 


0.94 


0.68 


0.39 


0.43 


0.5 


0.83 


0.63 


0.94 


0.47 


0.6 


0.97 


0.89 


0.83 


0.27 


0.7 


0.91 


0.88 


0.87 


0.40 


0.8 


0.64 


0.85 


0.80 


0.33 


0.9 


0.70 


0.37 


0.85 


0.40 



The hypothesis of hard truncation cannot be rejected for any of the four 
pieces. As expected, the p-values for piece 4 are lower than those for the 
other 3 pieces, but they are still rather high. 



Testing a stronger version of the hypothesis of hard truncation. The test 
statistics Z n (A) of Section 4.3 was computed and the corresponding p- values 
calculated for various values of e. These are listed in the following table. 



e 


piece 1 


piece 2 


piece 3 


piece 4 


0.1 


1.00 


1.00 


1.00 


1.00 


0.2 


1.00 


1.00 


1.00 


1.00 


0.3 


1.00 


1.00 


1.00 


0.91 


0.4 


1.00 


0.73 


1.00 


0.45 



It is clear that even the stronger version of the hypothesis of hard trun- 
cation cannot be rejected. 
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1.8 

1.6 

1.4- 

1.2- 



1 




0.2 0.4 0.6 D.B 1 1.2 1.4 1.6 1.8 2 2.2 



X10 7 

Figure 3. Data on Object Sizes 



5.2. Object Sizes. This data set contains 2.2 x 10 observations. It is 
plotted in Figure [3j It does not appear that the nature of the observations 
changes with time, so we applied our statistical tests to the entire data set. 
After running the Hill estimator with random k and parameters j3 and 7 as 
above, on the first half of the data set, we obtained a conservative upper 
bound on the value of the tail exponent a; this turned out to be A = 1.69. 
We used the second half of the Object Sizes data set to test for soft and 
hard truncations. 



Testing the hypothesis of soft truncation. We evaluated the test statistic 
Z n {Ai) of Section 4.1 for a range of values of A\ larger than A. The results 



are reported in the following table. 



A/At 


ZniAt) 


0.5 


1.75 


0.6 


2.32 


0.7 


3.09 


0.8 


4.10 


0.9 


5.42 


0.95 


6.23 



Comparing these with the corresponding quantiles (or their upper bounds) 
of Z(A/Ai), we see that the hypothesis of soft truncation cannot be rejected. 



Testing the hypothesis of hard truncation. We evaluated the test statistic 



Z n (A; 7) of Section 4.2 for various values of 7, and the obtained p- values are 



reported in the following table. 
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7 


p- value 


0.1 


0.50 


0.2 


0.36 


0.3 


0.73 


0.4 


0.77 


0.5 


0.95 


0.6 


0.94 


0.7 


0.94 


0.8 


0.97 


0.9 


0.72 



The null hypothesis of hard truncation cannot be rejected. 

Testing a stronger version of the hypothesis of hard truncation. We calcu- 



lated the test statistics Z n (A) of Section 4.3 for various values of e, and the 
p-values are given in the following table. 



e 


p- value 


0.1 


1.00 


0.2 


0.86 


0.3 


0.33 


0.4 


0.08 



The strengthened hypothesis of hard truncation becomes suspicious for e = 
0.4, but overall our statistical tests do not produce clear evidence of the level 
of truncation for the Object Sizes data set. 

Summary The results of this section, especially the lack of conclusion 
in a number of cases, point to some of the directions for future statistical 
work with truncated heavy tails. 

• More powerful tests are needed, together with estimates of their 
actual power. 

• What sample sizes are necessary for the asymptotic significance lev- 
els to be applicable? 

• The tests of Section [4] depend on tuning parameters A and 7. At 
present we do not have a clear picture of the role of these parameters, 
or how to select them in the "optimal" way. 

• What is the best way to select the parameters (3 and 7 in the sample- 



based number of the upper order statistics in (3.4) to use in Hill 
estimation? Can one combine estimation of the tail index with the 
testing for a truncation regime in a single procedure? 
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